^1 


AD-A263  31 


a  r ^  f 


»-•  V.  gj'.'.-''' 


4;' 

=  W#*. 


.  •  ■'■  ‘  .  .:  ■■'-■■  -v.r  ,  ■  -=  ---^.-  ■  f  jrr  ^./-*  -  v->r-»^  P"-'!- 

"  V  -:'  :*'  ■  ' -^  ■  <'■•/ A  :  •.  ■'  ■  ‘ 

.  ■•,  .,  i  .  ..  .  •’  •  .  ;'■■  ;«!.  '.*.  -'-  - 'r  i4‘ «t 'f*- --■* '-i^v ’. 

■  •  .•■  '...■’.■  ••  -  :-i  ..  .‘  .  V.’.U..V.  .•:<  \'-i*Ts  .'^•,.‘:;.j 


‘ ' '■^'i  '  / ‘'‘’’-i  i' W 

J',,''.‘-  ,■'■■■  ■''•■■■?>■■■''?!>■■.■.;;  ^■.^■■s' ‘  ■•■I-  ■  ' 

ntili  imiwfc  aiiiir  riiii  ■riiBlii  iiliyulh  IwgMltrjn rt f f  ■  li^liiM iii  ^tiiliiwi 


93-09073 


‘f'h  OSO 


-1 

1;K‘->  ■ 


-4 


GAO/GGD-91<42 


.i . 


/‘i'- 


- .  .. — ^ - 

\ United  States  v 

■^/\l  I  General  Accounting  OfHce 
<.AJ~  Washington,  D.C.  20548 


General  Government  Division 

B-234050 

March  11, 1991 


The  Honorable  William  L.  Clay 
Chairman,  Committee  on  Post  Office 
and  Civil  Service 
House  of  Representatives 

Dear  Mr.  Chairman: 


This  report  responds  to  the  Committee’s  request  that  we  review  the  Census  Bureau’s 
procedures  to  estimate  the  accuracy  of  the  census  counts  in  the  1988  dress  rehearsal,  the 
final  precensus  test.  The  basic  procedures  used  in  the  dress  rehearsal  are  currently  being 
used  in  the  Bureau’s  review  of  the  actual  1990  census.  The  report  focuses  on  the  post 
enumeration  survey,  which  is  the  key  census  activity  for  a  possible  adjustment.  The  report 
provides  information  to  illustrate  the  complex  and  difficult  nature  of  this  survey  and  related 
activities  that  the  Secretary  of  Commerce  will  use  in  making  his  decision  on  adjusting  the 
census  counts. 

As  arranged  with  your  office,  we  plan  no  further  distribution  of  the  report  until  30  days 
from  the  date  of  this  letter  unless  you  publicly  announce  its  contents  earlier.  At  that  time,  we 
will  send  copies  to  other  appropriate  congressional  committees;  the  Secretary  of  Commerce; 
the  Director,  Bureau  of  the  Census;  and  the  Director,  Office  of  Management  and  Budget. 
Copies  will  also  be  made  available  to  other  interested  parties  upon  request. 

The  major  contributors  to  this  report  are  listed  in  appendix  III.  If  you  have  any  questions 
concerning  this  report,  please  contact  me  on  275-8676. 

Sincerely  yours. 
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Elxecutive  Summary 

Purpose 

Decennial  census  results  are  used  for  such  important  purposes  as  reap¬ 
portioning  the  House  of  Representatives  and  redrawing  congressional, 
state,  and  municipal  legislative  district  lines.  However,  the  census  his¬ 
torically  has  undercounted  the  population,  especially  black  persons. 
According  to  the  estimate  most  often  cited  by  the  Bureau  of  the  Census, 
in  1980  the  white  and  nonblack  undercount  was  0.7  percent  while  the 
black  undercount  was  5.9  percent.  Such  undercounts  can  create  inequi¬ 
ties  in  political  representation  and  the  distribution  of  federal  funds. 
Because  of  public  concern  about  the  undercount,  the  Bureau  has  been 
studying  whether  adjustment  can  improve  census  counts. 

The  House  Committee  on  Post  Office  and  Civil  Service  asked  gao  to 
describe  the  Bureau’s  experience  with  the  1988  dress  rehearsal  post 
enumeration  survey  (*'  s)  and  to  identify  the  major  challenges  con¬ 
fronting  the  1990  PES.  The  pes  is  the  primary  methodology  the  Bureau  is 
using  as  a  basis  for  possibly  ac^usting  the  1990  census  counts;  it  com¬ 
pares  responses  from  households  interviewed  several  months  after 
Census  Day  with  census  questionnaires  from  the  same  housing  units  to 
determine  whether  each  person  was  correctly  counted,  missed,  or 
double-counted  in  the  census. 

Background 

Since  1950  the  Bureau  has  assessed  the  accuracy  of  its  census  counts. 
After  the  1980  census,  the  Bureau  intensified  its  research  into  a  possible 
means  of  adjusting  the  counts  in  the  1990  census.  The  effort  culminated 
in  the  final  test  of  procedures  in  the  dress  rehearsal  in  St.  Louis,  East 
Central  Missouri,  and  Elastem  Washington.  In  1987  the  Commerce 
Department  decided  that  it  would  not  ac^ust  the  1990  census  counts,  but 
a  lawsuit  filed  by  New  York  City  and  others  resulted  in  the  Depart¬ 
ment’s  agreement  to  reconsider  its  decision.  The  agreement,  incorpo¬ 
rated  in  a  court-approved  stipulation  and  order,  set  a  deadline  of  July 

15, 1991,  for  the  Department  to  publish  ai^usted  1990  census  data  if  the 
Secretary  decides  to  make  an  ac^ustment. 

Results  in  Brief 

The  dress  rehearsal  demonstrated  a  number  of  mgjor  hurdles  to  com¬ 
pleting  a  high-quality  1990  pes.  First,  it  confirmed  that  successfully 
matching  pes  and  census  records — the  foundation  upon  which 
undercount  estimates  are  based — is  extremely  difficult.  It  also  con¬ 
firmed  that  because  the  undercount  is  statistically  small,  even  a  small 
percentage  of  matching  errors  might  significantly  distort  undercount 
estimates.  Accurate  matching  will  be  an  even  greater  challenge  for  the 
1990  PES  than  in  the  dress  rehearsal  because  the  Bureau  must  implement 
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some  new  and  untested  procedures.  Further,  imlike  the  dress  rehearsal, 
the  1990  PES  is  nationwide  and  decentralized,  which  places  additional 
pressure  on  the  Bureau’s  ability  to  successfully  manage  matching  and 
other  PES  activities. 

The  dress  rehearsal  also  confirmed  that  because  of  the  difficulties  in 
successfully  completing  matching  and  other  operations,  careful,  thor¬ 
ough  assessments  of  the  pes  data  quality  are  vital.  The  Bureau’s  assess¬ 
ments  of  the  1990  PES  are  intended  to  show  the  degree  to  which  the  pes 
undercount  estimates  are  reliable.  As  such,  the  assessments  form  a  crit¬ 
ical  part  of  the  data  that  the  Secretary  will  need  to  make  an  adjustment 
decision. 

A  final  factor  complicating  adjustment  is  the  July  15, 1991,  decision 
deadline,  which  puts  severe  time  constraints  on  the  Bureau’s  efforts. 
According  to  both  the  Department  and  Bureau,  there  is  only  a  50- 
percent  chance  that  they  can  meet  the  agreed-upon  deadline. 


Principal  Findings 


latching  Poses  Major  dress  rehearsal  confirmed  that  matching  is  one  of  the  most  critical 

Ihallenfies  for  the  Bureau  difficult  aspects  of  the  pes.  gao  reviewed  the  Bureau’s  dress 

®  rehearsal  match  determinations  in  a  randomly  selected  375  (10  percent) 

of  the  households  in  which  one  or  more  persons  in  the  household  were 
identified  as  a  nonmatch  by  the  Bureau.  On  the  basis  of  the  review,  gao 
found  that  pes  or  census  data  may  be  incomplete,  inaccurate,  or  con¬ 
flicting.  For  example,  in  40  households  the  same  person  provided  dif¬ 
ferent  or  conflicting  information  in  the  census  and  pes  about  household 
members.  In  such  cases  it  can  be  difficult  to  determine  which  informa¬ 
tion  is  correct. 

Even  when  data  are  adequate,  pes  and  census  data  may  be  improperly 
matched.  Highly  accurate  matching  is  important  because  matching 
errors  in  even  a  small  percentage  of  cases  significantly  affect 
undercount  estimates.  For  example,  a  Bureau  evaluation  found  that 
without  matching  errors  in  the  dress  rehearsal,  the  estimated 
undercount  in  St.  Louis  would  have  been  4.6  percent  rather  than  5.8  per¬ 
cent,  and  in  East  Central  Missouri,  the  estimated  net  undercount  would 
have  been  3.5  percent,  not  4.7  percent. 
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The  nationwide  scope  and  the  higher  number  of  units  involved  in  the 
1990  PES  may  create  more  complications  than  in  the  dress  rehearsal  pes. 
For  1990,  the  Bureau  visited  and  completed  questionnaires  for  about 
170,000  housing  units  nationwide,  in  contrast  to  about  1 1,000  units  in 
the  1988  dress  rehearsal,  pes  activities  in  1990  employed  many  more 
people;  for  example,  as  many  as  850  clerks  were  employed  for  matching 
at  the  7  processing  offices  compared  with  50  clerks  at  the  1  processing 
office  in  the  dress  rehearsal.  Conflicting  match  decisions  were  adjudi¬ 
cated  by  as  many  as  12  pes  technicians  in  each  processing  office,  instead 
of  2  as  in  the  dress  rehearsal. 

Because  the  dress  rehearsal  was  done  in  a  limited  geographic  area,  the 
Bureau  was  unable  to  test  its  ability  to  follow  up  and  manage  matching 
efforts  for  individuals  who  moved  between  Census  Day  and  the  pes  from 
locations  covered  by  a  different  Bureau  processing  office.  Managing  and 
controlling  the  flow  of  documents  between  processing  offices  and  the 
additional  research  needed  to  attempt  to  match  individuals  who  moved 
also  complicated  1990  pes  matching  efforts.  (See  pp.  13-15.) 

Careful  Assessments 
Needed  to  Determine 
Quality  of  Undercount 
Estimates 

The  usefulness  of  the  undercount  estimates  depends  on  the  type  and 
amount  of  error  in  the  pes.  To  identify  such  error  in  the  undercount  esti¬ 
mates,  the  Bureau  plans  to  do  18  assessments  of  the  quality  of  the  1990 
pes.  The  most  important  assessment  will  evaluate  the  overall  effects  of 
errors  in  the  pes.  The  preliminary  results  of  the  dress  rehearsal  overall 
error  assessment  showed  that  the  estimated  true  net  undercount 
for  St.  Louis  was  between  4.0  and  8.5  percent.  Likewise,  the  true  net 
undercount  for  East  Central  Missouri  was  estimated  to  be  between  3.9 
and  10.9  percent.  (See  pp.  16-17.) 

PES  and  Related  Activities 
May  Not  Be  Completed  in 
Time  for  Adjustment 
Decision 

The  challenge  of  estimating  the  error  in  the  census  is  compoimded  by 
the  tight  schedule  required  to  meet  the  court-ordered  deadline  for  an 
acyustment  decision.  According  to  the  Bureau,  the  deadline  forced  it  to 
compress  the  original  pes  schedule,  thereby  putting  the  quality  of  the 

PES  at  risk.  For  example,  unlike  in  the  dress  rehearsal,  the  Bureau  col¬ 
lected  PES  data  at  the  same  time  it  was  collecting  some  basic  census  data. 
Overlapping  pes  and  census  data  collection  could  contribute  to  respon¬ 
dent  confusion  and  hostility,  which  hampers  data  collection  and  compli¬ 
cates  the  match  determinations.  The  Bureau’s  assessments  of  the  pes 
should  indicate  some  of  the  effects  of  respondent  confusion  and 
hostility. 

Page  4 


GAO/GGD-91-42  1990  Censua  Adjastment 


Executive  Summary 


The  Department  and  the  Bureau  have  consistently  stated  that  there  is 
only  a  50-percent  chance  that  adjustment  activities  will  be  completed  by 
the  agreed-upon  deadline.  To  date,  the  Bureau  generally  has  been  able  to 
accommodate  delays  in  the  pes  and  other  census-related  operations. 
However,  the  Bureau  still  faces  challenges  to  meeting  the  deadline.  For 
example,  pes  time  constraints  precluded  the  Bureau  from  incorporating 
into  the  usual  pes  process  all  persons  added  to  the  count  in  the  final 
phases  of  the  census.  These  persons  accounted  for  less  than  1  percent  of 
the  census  count.  However,  the  effect  that  not  fully  including  these  per¬ 
sons  in  the  pes  will  have  on  the  quality  of  the  undercount  estimates 
depends  on  the  degree  to  which  these  persons  are  concentrated  in  pes 
areas  and  whether  they  have  characteristics  different  from  other  per¬ 
sons  in  the  areas.  (See  p.  21.) 

The  Bureau  also  may  not  have  sufficient  time  to  finish  assessing  the 
quality  of  the  pes  by  the  adjustment  decision  deadline.  However,  it  plans 
to  provide  the  Department  with  preliminary  results  of  its  analyses  on  an 
ongoing  basis. 

The  Department  has  announced  that  the  Bureau  will  finish  the  pes  and 
its  assessments  even  if  the  studies  are  not  done  by  the  decision  deadline. 
GAO  agrees  that  all  aspects  of  the  pes  should  be  completed,  since 
undercount  estimates  of  the  1990  census  could  prove  valuable  for  the 
population  estimates  the  Bureau  prepares  between  censuses  and  for 
planning  future  censuses.  (See  p.  22.) 


gency  Comments 


GAO  obtained  the  views  of  responsible  Department  and  Bureau  officials 
on  its  findings  and  incorporated  their  comments  where  appropriate. 
Their  comments  were  mainly  technical  clarifications. 
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Chapter  1 _ 

Introduction 


Decennial  census  data  are  used  for  a  wide  variety  of  vitally  important 
purposes.  Foremost  among  these  are  reapportioning  the  House  of  Repre¬ 
sentatives  and  redrawing  congressional,  state,  and  municipal  legislative 
district  lines.  Census  data  are  also  used  extensively  during  the  following 
decade  for  allocating  billions  of  dollars  of  federal  and  state  funds  to 
political  subdivisions  and  by  both  the  public  and  private  sectors  in 
making  a  wide  range  of  policy,  economic,  and  operating  decisions. 

The  Census  Bureau  has  long  recognized  that  its  decennial  census  counts 
are  not  completely  accurate.  Some  people  are  missed  completely.  Others 
are  mistakenly  counted  twice;  for  example,  at  both  of  two  residences. 
Since  1950,  Bureau  evaluations  have  consistently  identified  a  net  census 
undercount  of  the  population.  The  undercount  rate  has  declined  with 
each  census,  but  still  remains  disproportionate  by  race.  The  most  often 
cited  estimates  by  the  Bureau  show  that  the  overall  net  census 
undercount  has  declined  from  4.4  percent  of  the  population  in  1950  to 
1.4  percent  in  the  1980  census.  However,  the  disproportionality  of  the 
undercount  remains.  For  example,  as  shown  in  figure  1,  blacks  histori¬ 
cally  have  been  undercounted  at  a  higher  rate  than  whites  and  other 
nonblacks.  In  1980,  the  Bureau’s  estimate  of  the  white  and  nonblack 
undercount  was  0.7  percent  while  the  black  undercount  was  5.9  percent. 
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ire  1:  Blacks  Have  a  Higher 
mated  Percent  Undercount  Than 
tes  and  Other  Nonblacks 


I  I  Whites  and  Other  Non-Blacks 
IH  Blacks 


Source:  Bureau  of  the  Census.  The  Coverage  of  Population  in  the  1960  Census  (PHC80-E4)  Feb.  1988. 


ljustment  Question 
IS  Been  Contentious 


Undercounts  that  are  not  equally  distributed  among  geographical  areas 
and  population  groups  can  create  inequities  in  political  representation 
and  the  distribution  of  public  funds.  However,  the  Bureau  has  never 
a(^justed  census  counts  to  compensate  for  the  historic  undercount.  For 
the  1980  census,  the  Bureau  planned  to  measure  the  errors  in  the  census 
counts  with  a  view  toward  possibly  adjusting  those  counts.  However, 
due  mainly  to  timing  concerns,  the  Bureau  significantly  reduced  its  orig¬ 
inal  plans  to  evaluate  the  accuracy  of  the  census.  Because  the  Bureau 
believed  that  there  were  limitations  in  the  accuracy  of  the  census  esti¬ 
mates  derived  from  its  reduced  evaluation  program,  an  acyustment 
using  that  data  could  have  added  more  error  to  the  counts  than  it  would 
have  corrected.  At  that  time  the  Bureau  believed  that  there  was  no  sta¬ 
tistically  defensible  method  of  distributing  the  national  level  of 
undercount  to  subnational  levels. 
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The  major  reduction  the  Bureau  made  in  its  1980  evaluation  effort  was 
the  elimination  of  a  planned  post  enumeration  survey  (pes)  of  about 
250,000  households.  The  Bureau  instead  completed  a  smaller  matching 
study.  A  PES  is  a  matching  study  in  which  an  independent  sample  of 
households  are  interviewed  several  months  after  Census  Day,  which  by 
law  is  April  1  in  the  census  year.  The  information  gathered  is  compared 
to  census  questionnaires  from  those  same  households  to  determine 
whether  each  person  was  correctly  counted,  missed,  or  double-counted 
in  the  census. 

The  question  of  whether  census  counts  should  be  adjusted  was  the  sub¬ 
ject  of  a  significant  amount  of  litigation  as  a  result  of  the  Bureau’s  deci¬ 
sion  not  to  adjust  the  1980  census.  According  to  the  Bureau’s  Office  of 
the  Chief  Counsel,  36  lawsuits  were  filed  against  the  Bureau  asking  that 
the  Bureau  be  required  to  adjust  the  1980  census  counts.  None  of  the 
cases  was  decided  against  the  Bureau. 

The  debate  over  adjustment  continued  when  the  Bureau  was  preparing 
for  the  1990  census.  After  the  1980  census,  the  Bureau  announced  an 
expanded  research  effort  to  develop  a  possible  means  of  adjusting  the 
1990  census  counts.  Initially  included  in  that  effort  were  proposals, 
which  were  not  fully  budgeted,  to  complete  a  1990  pes  of  300,000  house¬ 
holds.  However,  after  the  Department  of  Commerce  decided  in  1987  that 
it  would  not  adjust  the  1990  census  counts,  the  proposals  for  the 
300,000-household  pes  were  dropped  to  limit  costs.  The  Department 
believed  that  adjustment  might  introduce  more  error  than  it  solved  and 
might  divert  resources  needed  for  the  actual  census  enumeration.  The 
Department  instead  decided  to  do  a  1990  pes  of  150,000  housing  units  to 
evaluate  the  coverage  of  the  1990  census  and  assist  in  developing 
improved  census-taking  techniques. 

New  York  City,  other  localities,  and  some  public  interest  groups  subse¬ 
quently  filed  suit  against  the  Department  of  Commerce  and  others, 
asking  the  court  to  require  the  Department  to  do  a  full-scale  pes  and  to 
take  the  necessary  steps  to  correct  1990  census  counts  using  the  most 
accurate  correction  methods  available.'  On  July  17, 1989,  the  parties  to 
the  suit  agreed  in  a  court-approved  stipulation  and  order  that  all 
pending  motions  would  be  withdrawn,  and  the  Department  agreed  to 
vacate  its  earlier  decision  not  to  adjust  the  1990  census.  The  order  does 
not  preclude  the  plaintiffs’  rights  to  future  legal  actions  in  this  matter. 


'City  of  New  York  et  al.  v.  Department  of  Commerce  et  al.  Docket  No.  88,  Civ.  3474  (US.  Dist.  Ct., 
EbNY,  filed  Nov.  3, 19SS). 
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The  Department  also  agreed  to  do  a  pes  of  not  fewer  than  150,000 
households  (a  number  the  defendants  believed  was  sufficient)  and  other 
procedures  necessary  to  ensure  the  possibility  of  producing  adjusted 
counts  for  the  nation’s  approximately  100  million  households.  Such 
adijusted  counts  would  be  used  for  reapportionment,  redistricting,  and 
other  purposes.  As  required  by  the  court  order,  the  Department  devel¬ 
oped  guidelines  that  specified  the  grounds  upon  which  the  Secretary  of 
Commerce  will  base  an  adjustment  decision.  A  discussion  of  the  design 
and  mathematical  procedures  for  the  pes  is  included  in  appendix  I. 

The  Department  also  established  an  independent  panel  by  selecting  eight 
experts,  four  of  whom  were  recommended  by  the  plaintiffs,  to  make 
individual  recommendations  to  the  Secretary  on  whether  or  not  to 
adjust  census  counts.  According  to  the  court  order,  if  the  Secretary 
decides  to  make  an  adjustment,  corrected  counts  will  be  published  by 
July  15, 1991.  Both  the  Department  and  the  Bureau  have  publicly 
stated — as  recently  as  September  1990 — that  they  believe  there  is  only 
a  50-percent  chance  they  will  be  able  to  complete  the  pes  and  related 
adjustment  activities  by  the  agreed-upon  July  15, 1991,  deadline.  If  the 
Secretary  decides  not  to  adjust  the  counts,  the  Secretary  must  publish  a 
detailed  statement  explaining  that  decision  by  July  15, 1991. 


ethodologies  to 
itermine  Accuracy 
Census  Counts 


The  Bureau  traditionally  has  estimated  the  accuracy  of  the  census  by 
using  demographic  analysis,  a  comparison  between  the  census  enumera¬ 
tion  and  the  estimated  population  derived  from  such  sources  as  birth 
registrations,  death  records,  immigration  and  emigration  data.  Medicare 
files,  previous  censuses,  and  estimates  of  the  number  of  undocumented 
aliens.  The  net  undercount  estimates  from  the  1950  through  1980  cen¬ 
suses  shown  in  figure  1  are  based  on  demographic  analysis.  However, 
because  of  certain  limitations,  the  most  important  of  which  are  difficul¬ 
ties  in  measuring  subnational  migration,  demographic  analysis  has  not 
been  useful  for  identifying  census  errors  at  state  and  local  levels,  where 
accuracy  for  redrawing  congressional  and  state  legislative  districts  is 
critical.  Even  at  the  national  level,  difficulties  in  accurately  estimating 
the  number  of  undocumented  residents  has  reduced  the  utility  of  the 
demographic  analysis. 

Demographic  analyses  are  helpful  in  measuring  the  error  in  the  pes  and 
similar  methodologies  that  rely  on  matching  operations.  Demographic 
analyses  assist  in  identifying  the  amount  of  correlation  bias,  the  loss  of 
independence  between  the  census  enumeration  and  the  pes  that  exists 
because  persons  who  are  missed  in  the  census  are  also  more  likely  to  be 
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missed  in  the  pes.  For  example,  some  individuals  simply  do  not  want  to 
be  counted  and  will  therefore  avoid  both  the  census  and  the  pes.  Thus, 
the  undercounts  identified  as  part  of  the  pes  will  tend  to  underestimate 
the  true  undercount.  (See  app.  I,  especially  pp.  26-27  for  a  discussion  of 
correlation  bias.)  The  Bureau  currently  is  exploring  methodologies  that 
would  enable  it  to  combine  the  results  of  the  pes,  its  demographic  anal¬ 
ysis,  and  the  census  enumeration  to  develop  adjusted  1990  census 
counts. 

Despite  correlation  bias,  matching  studies  such  as  the  pes  are  used  to 
identify  census  population  errors  at  subnational  levels.  The  Bureau  con¬ 
siders  the  PES  to  be  the  key  component  of  its  overall  evaluation  strategy 
to  measure  census  coverage  and  possibly  ac(just  the  census  counts. 

The  Bureau  tested  pes  procedures  during  the  census  test  cycle  for  the 
1990  census,  including  the  1988  dress  rehearsal.  The  dress  rehearsal, 
which  was  done  in  three  locations  (St.  Louis,  East  Central  Missouri,  and 
Eastern  Washington)  was  the  Bureau’s  last  opportunity  to  test  its  census 
procedures.  Only  minor  modifications  are  supposed  to  be  made  subse¬ 
quent  to  the  dress  rehearsal. 

The  dress  rehearsal  pes  was  generally  done  at  the  time  when  the  Depart¬ 
ment  had  decided  not  to  ac^ust  the  counts.  Consequently,  it  was  not 
done  on  a  production  schedule  required  for  meeting  a  deadline.  As  a 
result,  the  Bureau  was  forced  to  make  major  changes  in  its  i>ES  1990 
operations  in  an  attempt  to  accommodate  the  a((justment  decision  dead¬ 
line.  These  changes  are  discussed  in  chapter  3. 


Objectives,  Scope,  and 
Methodology 


Because  of  the  controversy  surrounding  the  possible  a((justment  of  the 
1990  census,  the  House  Committee  on  Post  Office  and  Civil  Service 
asked  us  to  monitor  dress  rehearsal  pes  operations  and  procedures 
related  to  census  adjustment.  Our  overall  objective  was  to  illustrate  the 
complex  and  difficult  nature  of  the  pes  by  describing  the  Bureau’s  expe¬ 
rience  with  the  PES  dress  rehearsal  and  identifying  the  challenges  con¬ 
fronting  the  Bureau  for  1990.  Specifically,  our  objectives  were  to  review 
( 1 )  the  procedures,  operations,  timing,  and  results  of  the  dress  rehearsal 
PES  and  (2)  the  procedures  and  schedule  for  the  1990  pes. 


We  did  not  assess  the  extent  to  which  the  Bureau’s  decision  to  do  a  pes 
with  the  current  sample  size  of  150,000  households  as  opposed  to  the 
300,000-household  pes  that  was  considered  earlier  in  the  decade  will 
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limit  the  utility  of  the  undercount  estimates  for  the  purposes  of  a  pos¬ 
sible  adjustment.  Some  of  the  implications  of  the  Bureau’s  decision  to  do 
the  smaller  pes  will  be  known  when  the  pes  and  evaluations  of  its  errors 
are  complete. 

To  meet  our  objectives,  we  reviewed  the  schedule  of  pes  operations  and 
activities  for  the  1990  census  and  compared  these  to  the  Bureau’s  per¬ 
formance  during  the  dress  rehearsal  in  1988.  We  examined  the  Bureau’s 
procedural  manuals  to  determine  the  methods  the  Bureau  used  for 
matching,  time  schedules,  observation  reports,  and  internal  memoranda, 
and  also  reviewed  other  documents,  including  1990  pes  progress  reports 
through  December  1990.  We  reviewed  the  dress  rehearsal  pes  results, 
undercount  estimates,  the  Bureau’s  evaluations  of  those  estimates,  and 
prior  GAO  work.  We  also  interviewed  Bureau  officials  responsible  for 
designing,  managing,  and  evaluating  pes  operations  in  1988  and  in  the 
1990  census. 

To  evaluate  dress  rehearsal  pes  efforts  to  match  individuals  identified  in 
the  PES  with  the  census,  we  reviewed  the  match  decisions  for  1,215  per¬ 
sons  in  375  households.  The  household  cases  were  selected  by  randomly 
sampling  10  percent  of  all  households  in  which  one  or  more  persons 
were  not  matched  in  the  comparison  of  PES-  and  census-counted  persons 
in  the  three  dress  rehearsal  sites.  The  results  of  our  sample  cannot  be 
projected  for  the  dress  rehearsal. 

Our  audit  work  was  done  between  July  1989  and  December  1990  at 
Bureau  headquarters  in  Suitland,  Maryland,  and  in  the  dress  rehearsal 
processing  office  in  Kansas  City,  Missouri,  in  accordance  with  generally 
accepted  government  auditing  standards.  We  did  not  obtain  official 
agency  comments  on  this  report.  However,  we  obtained  the  views  of 
responsible  Department  and  Bureau  officials  and  incorporated  their 
comments  where  appropriate.  Their  comments  were  mainly  technical 
clarifications. 
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The  Bureau  faces  formidable  challenges  to  complete  an  accurate  pes  and 
associated  activities.  Like  the  census,  the  pes  also  is  subject  to  error,  and 
even  a  relatively  small  percentage  of  errors  can  have  significant  effects 
on  the  estimation  of  the  over-  or  undercount  rates.  Careful  and  thorough 
evaluations  are  essential  to  measure  the  amount  of  error  in  the  pes  and 
the  degree  to  which  the  Secretary  can  have  crnfidence  in  the  over-  or 
undercount  estimates  when  making  an  adjustment  decision. 


Matching  Is  Difficult 
but  Vital  for  Success 
of  PES 


Matching  individuals  identified  in  the  pes  with  census  records  by  using 
certain  key  data  such  as  address,  name,  sex,  age,  and  race  is  one  of  the 
most  crucial  and  difficult  aspects  of  the  pes.  Generally,  incorrect 
matching  determinations  can  result  from  two  sources;  errors  caused  by 
incomplete,  inaccurate,  or  conflicting  data  and  errors  where  a  poor 
match  decision  was  made  even  though  the  data  were  sufficient.  In  either 
instance,  errors  contribute  to  uncertainty  about  the  accuracy  of  the  pes 
estimates  of  census  over-  or  undercount  rates. 


Inaccurate,  Incomplete,  or 
Conflicting  Data 
Complicate  Matching 
Efforts 


One  challenge  confronting  the  Bureau  is  to  obtain  accurate  and  complete 
information  to  facilitate  matching  efforts.  In  our  review  of  match  deter¬ 
minations  in  the  pes  dress  rehearsal,  examples  of  which  are  included  in 
appendix  II,  we  found  that  in  about  19  percent  of  the  match  determina¬ 
tions  either  the  census  or  the  pes  had  insufficient  information  to  confi¬ 
dently  determine  if  each  person  was  counted  or  not  counted  in  the 
census.'  As  a  result,  the  Bureau  was  forced  to  make  some  difficult  and 
potentially  arguable  match  determinations.  For  example: 


•  In  40  of  the  375  households  we  reviewed,  the  same  person  interviewed 
as  part  of  the  census  and  the  pes  provided  conflicting  information  about 
the  household  members.  In  a  few  other  households,  different  but  still 
knowledgeable  respondents  also  provided  different  and  conflicting 
information  about  where  household  members  were  living  on  Census 
Day.  The  time  interval  between  Census  Day  and  the  pes  interview  can 
introduce  a  problem  of  correct  recall.  In  such  situations,  it  can  be  diffi¬ 
cult  to  determine  which  information  is  correct.  Consequently,  the  degree 
to  which  the  resulting  match  determinations  are  correct  is  open  to  con¬ 
flicting  interpretation. 


'This  calculation  applies  to  households  who  did  not  move  between  the  dress  rehearsal  Census  Day 
and  the  PES.  Our  sample  of  1,215  persons  included  about  100  movers,  for  which  complete  informa¬ 
tion  could  not  be  obtained  because  of  the  limited  geographical  area  covered  in  the  dress  rehearsal. 
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The  Bureau  estimated  that  if  dress  rehearsal  Census  Day  address 
errors — the  inaccurate  reporting  during  the  pes  of  a  person’s  Census 
Day  address — had  not  occurred,  the  estimated  undercount  would  have 
been  reduced  markedly.  Assuming  no  other  sources  of  error,  the 
undercount  estimates  for  St.  Louis  would  have  been  4.2  percent  rather 
than  5.8  percent,  a  difference  of  about  28  percent.  Likewise  for  East 
Central  Missouri,  the  estimated  undercount  would  have  been  4  percent 
as  opposed  to  4.7  percent,  a  difference  of  about  15  percent.^  (Data  for 
Eastern  Washington  were  not  developed  due  to  the  limited  amount  of 
data  from  that  test  site.) 

•  For  18  of  the  375  households,  including  5  cited  above,  the  census  enu¬ 
merators  and/or  the  pes  interviewers  noted  that  the  respondent  was 
hostile  and  either  reluctantly  provided  information  about  the  household 
or  refused  to  provide  information.  Respondent  hostility  is  a  m^or  factor 
that  can  cause  the  census  and/or  the  pes  to  fail  to  count  persons  or  to 
improperly  classify  persons  as  matched,  correctly  enumerated,  not 
matched,  or  unresolved. 

In  instances  where  the  Bureau  is  unable  to  make  a  decision,  the  indi¬ 
vidual  is  classified  as  “unresolved.”  During  the  dress  rehearsal,  the  rate 
of  unresolved  cases  ranged  from  1.7  percent  to  3.8  percent  of  all  match 
cases  across  the  three  test  sites.  Unresolved  cases  later  are  changed  to 
matched  or  unmatched  using  a  statistical  procedure  known  as  imputa¬ 
tion.  Imputation  assigns  a  match  status  based  on  an  examination  of  the 
results  of  similar  cases  where  the  Bureau  was  able  to  make  a  determina¬ 
tion.  While  some  imputation  is  unavoidable,  it  introduces  imprecision 
into  estimates  of  census  over-  or  undercount  rates. 

The  Bureau’s  decision  to  reduce  1990  census  questionnaire  follow-up 
efforts  may  have  complicated  the  pes  process  and  may  increase  the 
uncertainty  of  the  over-  or  undercount  estimates.  To  limit  census  costs, 
the  Bureau  decided  in  1989  to  reduce  its  follow-up  efforts  to  a  10  per¬ 
cent  sample  of  those  short-form  questionnaires,  which  most  households 
received,  that  failed  the  Bureau’s  edit  requirements  for  certain  census 
content  data,  such  as  age,  sex,  and  race.  To  the  extent  such  households 
are  part  of  the  pes  sample,  the  reduced  census  follow-up  will  make  pes 
matching  more  difficult  because  the  Bureau  did  not  obtain  the  charac¬ 
teristics  of  some  of  the  persons  counted  in  the  census.  Because  of  the 
reduced  follow-up,  the  Bureau  will  do  more  imputations,  in  which  other 


^These  differences  may  be  reduced  in  the  actual  census  because  the  Bureau  will  attempt  to  validate 
the  address  of  movers  since  the  entire  country  is  included  in  the  census. 
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persons’  characteristics  are  substituted  for  the  missing  data.  Using  more 
imputations  increases  the  uncertainty  of  the  pes  projections  of  over-  or 
undercount  estimates  for  the  various  population  groups. 


Matching  Errors 
Significantly  Affect 
Undercount  Estimates 


A  second  challenge  for  the  Bureau  is  to  control  the  number  of  matching 
errors  resulting  from  an  incorrect  determination  even  when  sufficient 
information  is  available.  A  Bureau  evaluation  of  the  dress  rehearsal  pes 
found  that  the  Bureau  should  have  matched  more  persons.  According  to 
this  evaluation,  if  matching  errors  were  eliminated,  the  undercount  for 
St.  Louis  would  have  been  4.6  percent  instead  of  the  estimated  5.8  per¬ 
cent,  a  reduction  of  21  percent,  and  the  undercount  for  East  Central  Mis¬ 
souri  would  have  been  3.5  percent  rather  than  the  estimated  4.7  percent, 
a  reduction  of  26  percent. 

The  nationwide  scope  and  the  higher  number  of  units  involved  in  the 
1990  PES  will  challenge  the  Bureau’s  ability  to  maintain  the  same  level  of 
management  oversight  and  uniformity  in  making  matching  determina¬ 
tions  as  in  the  dress  rehearsal  pes.  For  1990,  pes  enumerators  visited  and 
completed  questionnaires  for  about  170,000  housing  units  nationwide, 
about  a  15-fold  increase  over  the  1 1,000  units  in  the  1988  dress 
rehearsal.^  pes  activities  in  1990  employed  many  more  people;  for 
example,  as  many  as  850  matching  clerks  were  employed  at  the  7 
processing  offices  compared  with  about  50  clerks  at  the  1  processing 
office  in  the  dress  rehearsal — a  17-fold  increase  in  matching  clerks. 
Ac^judication  of  conflicting  match  decisions  were  handled  by  as  many  as 
12  PES  technicians  in  each  processing  office,  instead  of  2  as  in  the  dress 
rehearsal. 


Careful  Assessments 
Elssential  for 
Af^ustment  Decision 


The  difficulties  in  successfully  completing  the  pes  and  the  fact  that  the 
PES,  like  all  surveys,  is  subject  to  a  variety  of  errors,  underscores  the 
importance  of  completing  sound  and  careful  assessments  of  the  quality 
of  the  PES.  The  error  in  the  pes  must  be  evaluated  before  detennining  if 
the  pes  results  should  be  used  for  adjusting  the  census  counts.  The  value 
of  the  PES  estimates  will  be  appreciably  reduced  if  the  error  in  those 
estimates  is  considered  significant.  Thus,  assessments  of  the  pes  form  a 
critical  part  of  the  data  the  Secretary  of  Commerce  will  need  to  make 
the  ac^justment  decision. 


^About  20,000  of  these  housing  units  were  vacant.  As  a  result,  about  150,000  PES  household  ques¬ 
tionnaires  were  prepared. 
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The  Bureau  plans  to  study  its  1990  pes  and  demographic  analysis  to 
determine  the  quality  of  those  efforts  and  the  degree  to  which  estimates 
of  the  census  over-  or  undercount  rates  appear  to  be  accurate  and  reli¬ 
able.  The  Bureau  plans  to  do  29  studies — 18  to  evaluate  the  effects  of 
error  that  can  occur  in  the  pes  and  1 1  to  evaluate  the  Bureau’s  1990 
demographic  analysis.  The  potential  pes  errors  to  be  measured  include 
missing  data,  erroneously  reported  Census  Day  addresses,  data  falsifica¬ 
tion,  errors  in  matching  census  and  pes  records,  and  other  sources  of 
error.  The  studies  of  demographic  analysis  generally  will  evaluate  the 
effects  of  errors  in  the  data  used  in  that  methodology  to  independently 
estimate  the  population  for  1990.  We  currently  are  reviewing  the 
Bureau’s  1990  pes  and  demographic  analysis  assessments  as  part  of  a 
separate  effort. 

The  single  most  important  evaluation  that  the  Bureau  has  scheduled  will 
assess  the  overall  effects  of  errors  in  the  pes.  Some  types  of  errors — for 
example,  the  matching  errors  discussed  on  page  16 — generally  con¬ 
tribute  to  an  overestimation  of  the  true  undercount.  Other  types  of 
errors,  for  example  correlation  bias,  lead  to  an  underestimation  of  the 
true  undercount.  Total  error  estimation  combines  several  key  sources  of 
error  to  show  whether  the  coverage  error  estimates  derived  from  the  pes 
are  sufficiently  accurate  to  be  used  for  acyusting  the  1990  census  for  an 
expected  net  undercount,  especially  at  geographic  levels  used  for  legisla¬ 
tive  redistricting.  For  1990, 9  of  the  Bureau’s  18  studies  to  evaluate  the 
quality  of  the  pes  will  be  used  to  form  the  total  error  estimate. 

Preliminary  results  of  the  total  error  study  that  the  Bureau  did  as  part 
of  the  dress  rehearsal  pes,  given  certain  assumptions,  estimated  the  true 
net  undercount  for  St.  Louis  as  between  4.0  percent  and  8.5  percent  with 
95-percent  confidence.^  Likewise,  the  true  net  undercount  for  East  Cen¬ 
tral  Missouri  was  estimated  to  be  between  3.9  percent  and  10.9  percent, 
with  95-percent  confidence.  No  error  estimates  were  developed  for  the 
Eastern  Washington  area  because  that  site  did  not  provide  a  sufficient 
amount  of  data  for  this  type  of  analysis. 


C!onclusion 


We  found  as  part  of  our  review  of  selected  dress  rehearsal  pes  cases  that 
incomplete,  inaccurate,  or  conflicting  data  can  make  matching  extremely 


^The  96-percent  confidence  level  means  that,  in  a  Itmg  series  of  replications  of  the  PES,  the  true  net 
undercount  rate  would  fall  between  lower  and  upper  limits  constructed  in  the  same  way  about  95 
percent  of  the  time.  For  detailed  results  of  the  dress  rehearsal  total  error  study,  see  Total  Error  in 
PES  Estimates  of  Population:  The  Dress  Rehearsal  Census  of  1988,  Mary  H.  Mulry  and  Bruce  D. 
Spencer,  (Draft)  March  16, 1990. 
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difficult  and  leave  some  determinations  open  to  conflicting  interpreta¬ 
tion.  Other  errors,  even  when  sufficient  data  were  available,  were  made 
in  the  matclung  operation.  Due  to  the  nationwide  scope  and  much  larger 
number  of  units  involved,  successful  matching  in  the  1990  pes  is  an  even 
greater  challenge  for  the  Bureau  than  it  was  during  the  dress  rehearsal. 
As  a  result,  the  pes  generally,  and  matching  efforts  in  particular,  require 
complete,  careful  scrutiny  and  thorough  assessment  to  ensure  that  the 
Secretary  is  provided  with  key  data  on  the  extent  to  which  the  over-  or 
undercount  estimates  are  accurate  and  reliable.  The  Bureau  plans  to  do 
18  studies  to  measure  error  in  the  pes. 
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The  Department  and  the  Bureau  have  repeatedly  commented  on  the  dif¬ 
ficulties  of  completing  all  the  activities  in  the  adjustment  methodology 
before  the  deadline.  They  have  consistently  said — as  recently  as  Sep¬ 
tember  1990 — that  there  is  only  a  50-percent  chance  that  the  pes  and 
related  activities  will  be  completed  by  July  15, 1991.  In  testimony  in 
January  1990,  we  also  discussed  the  difBculties  confronting  the  Bureau 
in  finishing  by  the  deadline.*  However,  as  noted  by  the  Bureau  Director, 
the  Bureau  has  been  able  to  accommodate  delays  in  the  pes  and  other 
census  related  operations.  Nonetheless,  difficult  aspects  of  the  pes  still 
confront  the  Bureau  in  the  remaining  months  before  the  adjustment 
decision  deadline. 


Acljustment  Deadline 
Requires  Risky  PES 
Time  Schedule 


In  October  1989,  the  Bureau  submitted  a  plan,  including  timetables,  to 
the  Department  for  doing  the  activities  necessary  to  meet  the  July  15, 
1991,  adjustment  decision  deadline.  The  Bureau  noted  it  was  forced  to 
compress  the  pes  schedule  and  establish  time  frames  that  put  the  quality 
of  the  PES  at  risk. 


The  collection  of  pes  data  while  the  actual  census  was  still  going  on  is 
one  risk,  because  overlapping  data  collection  could  compromise  the 
quality  of  both  the  pes  and  the  census.  In  the  dress  rehearsal,  pes  data 
gathering  was  not  started  until  almost  all  the  basic  census  data  collec¬ 
tion  had  been  completed.  However,  to  meet  1990  pes  deadlines,  the 
Bureau  scheduled  pes  field  interviewing  to  start  on  June  25, 1990 — 
before  all  basic  census  collection  activities  were  completed.  As  a  result, 
the  Bureau  interviewed  pes  households  while  late  census  data  collection 
was  still  underway.  These  late  census  data  collection  efforts,  scheduled 
to  begin  on  June  28, 1990,  included  (1)  checking  on  the  status  of  the 
approximately  12  million  housing  units  that  were  classified  as  vacant  or 
nonexistent  during  earlier  operations,  (2)  contacting  some  households 
that  did  not  provide  complete  questionnaire  information,  and  (3)  con¬ 
tacting  a  residual  number  of  households  that  did  not  respond  to  ques¬ 
tionnaires  and  were  not  completed  in  the  Bureau’s  initial  follow-up  of 
households  that  did  not  mail  back  the  census  questionnaires. 

In  practice,  enumerators  who  worked  on  the  census  and  became  aware 
of  the  fact  that  their  areas  were  in  the  pes  blocks  may  have  ac^justed 
their  efforts  accordingly.  While  the  effect  on  data  quality  is  not  possible 
to  quantify,  we  believe  the  enumerators  may  have  worked  more  or  less 


‘Critical  Issues  for  Census  Ac^iustment:  Completing  Post  Enumeration  Survey  on  Time  While  Pro¬ 
tecting  Data  Quality  (GAO/T.W&-^-16,  Jw.  3071990). 
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diligently  because  they  knew  their  results  would  be  systematically 
checked.  If  so,  there  would  be  at  least  two  adverse  effects  on  pes  results. 
First,  the  census  results  obtained  from  pes  blocks  would  not  be  represen¬ 
tative  of  the  national  census  coverage  if  the  enumerators  deviated  from 
their  normal  data  collection  efforts.  Second,  respondents  who  were 
annoyed  or  confused  by  being  asked  to  give  information  in  two  Census 
Bureau  inquiries  at  about  the  same  time  might  either  have  provided 
poorer  quality  data  or  refused  to  cooperate  in  one  or  both  inquiries.  The 
latter  effect  would  compromise  the  critical  assumption  of  the  Bureau’s 
technique  for  estimating  net  census  undercount  (see  app.  I),  namely  that 
coverage  in  the  census  is  independent  of  coverage  in  the  pes.  The 
Bureau’s  assessments  of  the  pes  will  measure  some  of  the  effects  of 
overlapping  census  and  pes  data  collection. 

Another  schedule  compression  the  Bureau  made  in  an  attempt  to  meet 
the  court-ordered  deadline  was  to  reduce  the  time  allotted  for  the  major 
clerical  matching  operation  from  10  to  7  weeks.  The  Bureau  noted  that 
the  reduced  time  for  clerical  matching  could  place  the  quality  of  pes  data 
at  risk.  The  Bureau  was  able  to  complete  virtually  all  clerical  matching 
in  the  time  allowed.  However,  it  will  not  be  clear  until  the  evaluations  of 
PES  matching  efforts  are  completed  in  May  1991  what,  if  any,  effect  the 
compressed  schedule  had  on  the  quality  of  the  matching  process. 

After  the  initial  matching  operations  were  completed,  some  cases  needed 
follow-up  interviewing  for  final  resolution.  The  Bureau  had  scheduled 
this  interviewing  for  the  period  November  9  through  December  24, 

1990.  With  the  exception  of  a  few  outstanding  cases  in  the  Detroit  and 
Los  Angeles  regions,  pes  follow-up  was  completed  by  December  28, 

1990.  The  Bureau  completed  the  remaining  cases  during  early  January 

1991.  During  the  dress  rehearsal  pes,  the  Bureau  anticipated  problems  in 
staffing  and  finding  people  at  home  during  the  December  holiday  period 
and  deferred  its  follow-up  efforts  until  January,  an  option  that  was  not 
realistic  under  the  1990  pes  time  constraints. 


New  and  Untested 
Procedures  Pose 
Additional  Risk  to  PES 
Schedule 


Some  of  the  major  1990  pes  procedures  have  not  previously  been  tested, 
which  contributes  to  the  uncertainty  that  the  Bureau  will  be  able  to 
meet  the  acyustment  deadline.  For  example,  the  Bureau’s  ability  to 
follow  up  and  manage  matching  efforts  for  individuals  who  moved 
between  Census  Day  and  the  pes  from  locations  covered  by  a  different 
Bureau  processing  office  generally  had  not  been  tested.  During  the  dress 
rehearsal,  about  10  percent  of  persons  in  the  pes  sample  moved  in  the  4- 
month  period  between  the  census  and  the  pes.  Follow-up  on  persons  who 
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moved  between  the  census  and  the  pes  was  limited  in  the  dress  rehearsal 
because  of  the  small  geographic  area  covered  by  the  test. 

This  was  not  the  case  with  the  1990  pes.  The  Bureau’s  ability  to  control 
the  flow  of  documents  between  the  seven  processing  offices  was  not 
tested  in  the  dress  rehearsal  because  only  one  processing  office  was 
used.  Matching  individuals  found  in  the  1990  pes  who  had  moved  since 
Census  Day  were  difficult  because  of  the  additional  research  and  logis¬ 
tics  involved  in  transmitting  documents  between  processing  offices.  The 
Bureau’s  assessment  of  its  pes  matching  efforts  should  provide  an 
important  indication  of  the  degree  to  which  the  Bureau  successfully 
managed  mover  matching. 


Another  untested  aspect  of  the  1990  pes  is  the  computer  software 
needed  to  accommodate  the  considerably  larger  pes  files  from  seven 
processing  offices,  compared  to  dress  rehearsal  files  in  one  office.  In  late 
1990,  the  Bureau  was  still  developing  critical  software  for  processing 
and  estimation,  and  plans  to  test  it  shortly  before  the  time  of  the  first 
production  run.  This  will  provide  little  opportunity  for  changes  should 
the  software  require  significant  corrections. 


In  the  past,  computer  software  problems  have  delayed  pes  operations. 
For  example,  during  the  1986  Los  Angeles  test  census  pes,  major  delays 
occurred  in  estimating  the  census  errors  and  ac^justing  the  census  files.^ 
The  Bureau  attributed  these  delays,  in  part,  to  (1)  problems  with  un¬ 
tested  software  and  (2)  keying  errors  in  preparing  a  computer  file  of  the 
matching  results.  Preparation  of  the  pes  results  necessary  for  estimation 
was  initially  scheduled  for  1  day,  but  subsequently  took  about  6  weeks 
to  complete.  In  addition,  problems  in  a  computer  program  for  keying  in 
PES  questionnaires  delayed  the  dress  rehearsal  pes  about  1  month. 


Late  Additions  to  1990 
Census  Counts 
Complicated  Matching 
smd  Follow-Up  Efforts 


The  need  to  rematch  census  and  pes  files  to  include  late  additions  to  the 
1990  census  counts  further  complicated  pes  operations.  In  the  dress 
rehearsal,  late  additions  to  the  census  count  were  not  systematically 
incorporated  into  the  pes.  For  the  1990  census,  persons  were  added  to 
the  counts  as  a  result  of  procedures  done  late  in  the  enumeration  phase 
to  improve  the  completeness  and  accuracy  of  the  census.  For  example, 
the  Bureau  identified  about  321,000  potential  additions  to  its  housing 
unit  list  as  a  result  of  a  national  recanvassing  effort,  known  as  the 


^See  Decennial  Census;  Coveraf?e  Evaluation  and  Adjustment  Activities  (GAO/GGD-87-99FS,  July 
1987). 
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Housing  Coverage  Check,  which  was  not  completed  until  October.  The 
Housing  Coverage  Check,  which  was  not  part  of  the  Bureau’s  plans  for 
the  census,  was  implemented  in  July  1990,  after  the  Bureau’s  research 
indicated  that  it  had  missed  housing  units  when  it  developed  the  census 
address  list. 

To  the  extent  that  these  late  additions  were  in  PES  areas,  they  needed  to 
be  incorporated  into  the  census  files  used  in  the  pes  after  the  initial 
matching  files  were  developed  and  most  matching  operations  were  com¬ 
pleted.  However,  not  all  of  these  late  census  data  were  available  in  time 
for  complete  matching.  The  Bureau  decided  that  late  census  data  added 
in  late  November  and  December  would  not  be  incorporated  into  the 
usual  PES  process.  Because  of  time  constraints,  field  follow-up  inter¬ 
viewing  was  not  done.  The  number  of  persons  added  to  the  census  as  a 
result  of  the  late  census  activities  but  not  incorporated  into  the  complete 
matching  operation  amounted  to  less  than  1  percent  of  the  persons 
counted  in  the  census.  However,  their  influence  on  the  quality  of  the 
undercount  estimates  will  depend  on  whether  they  were  concentrated  in 
PES  areas  and  whether  these  persons  have  characteristics  different  from 
other  persons  in  the  area. 


Time  Allowed  to 
Assess  Quality  of 
Adjustment  Efforts 
May  Be  Insufficient 


The  Bureau  has  identified  the  need  to  complete  its  assessments  of  the 
PES  by  July  15, 1991,  as  a  m^jor  risk  to  meeting  the  adjustment  decision 
deadline.  As  we  noted  in  chapter  2,  such  assessments  form  a  critical  part 
of  the  data  the  Secretary  of  Commerce  will  need  to  decide  whether  or 
not  to  adjust  the  census  counts.  The  Bureau’s  schedule  requires  adjust¬ 
ment  activities,  including  most  planned  assessments  of  the  quality  of  the 
PES,  to  be  completed  by  mid-May  1991.  The  Bureau  has  noted  that  the 
scheduled  dates  for  the  assessments  depend  on  the  planned  schedule  of 
activities  for  the  1990  census  and  pes  being  met. 

Unfortunately,  the  total  error  study  depends  on  the  timely  completion  of 
the  supporting  assessments  and  therefore  delays  in  finalizing  the  sup¬ 
porting  assessments  could  delay  the  total  error  study.  At  the  time  we 
completed  our  field  work  in  December  1990,  the  Bureau’s  schedule 
called  for  it  to  complete  the  total  error  study  at  about  the  time  of  the 
adjustment  deadline.  However,  in  February  1991,  the  Bureau  issued  a 
revised  schedule  that  indicated  the  total  error  study  would  be  completed 
by  early  June  1991.  In  either  case,  these  dates  may  be  overly  optimistic. 
Data  problems  could  affect  the  completion  date  of  the  Bureau’s  analysis. 
For  the  dress  rehearsal,  the  draft  of  the  results  of  the  total  error  study 
was  dated  March  16, 1990,  about  2  years  after  the  dress  rehearsal 
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Census  Day.  As  of  December  1990,  the  dress  rehearsal  total  error  study 
had  not  been  finalized  due  to  higher  priorities,  according  to  the  Bureau. 
The  Bureau  plans  to  provide  to  the  Department  preliminary  results  of 
its  1990  PES  analysis  on  an  ongoing  basis. 

The  Department  stated  in  its  guidelines  for  a  possible  adjustment  that 
the  assessments  of  the  quality  of  the  pes  and  related  ai^ustment  activi¬ 
ties  will  be  completed  even  if  they  cannot  be  done  until  after  July  15, 
1991.  We  believe  such  data  could  be  useful  in  determining  whether  to 
ac^ust  other  data  series,  such  as  estimates  prepared  between  censuses. 
For  example,  as  we  recently  reported,®  in  fiscal  year  1989,  93  federal 
formula  programs  involving  funds  totaling  $27.5  billion  used  Census 
Bureau  population  data,  in  whole  or  in  part,  to  determine  program  eligi¬ 
bility  or  to  distribute  funds  to  state  and  local  governments.  Of  these  93 
programs,  48  used  current  population  estimates  to  distribute  $10.1  bil¬ 
lion,  and  45  used  1980  decennial  census  population  data  to  distribute 
$17.4  billion. 

One  of  the  continuing  objectives  of  the  Bureau’s  evaluation  efforts  since 
1950  has  been  to  develop  improved  census-taking  techniques.  Com¬ 
pleting  the  assessments  of  the  1990  pes  will  aid  the  Bureau  in  identi¬ 
fying  possible  future  improvements  in  the  census,  including  ac^ustment 
methodologies. 


Conclusions 


The  July  15, 1991,  deadline  for  making  an  acljustment  decision  forced 
the  Bureau  to  implement  a  revised  pes  schedule  that  contains  risks  to 
the  quality  of  pes  data.  The  Bureau’s  need  to  employ  overlapping  census 
and  PES  data  collection  activities,  use  untested  pes  procedures,  compress 
the  time  allowed  for  matching,  exclude  some  late  census  data  from  being 
incorporated  into  the  usual  pes  process,  complete  pes  field  follow-up 
efforts  during  the  December  holiday  season,  and  thoroughly  assess  the 
quality  of  the  pes  and  related  acyustment  effort  before  the  deadline  all 
pose  major  risks  to  the  quality  of  the  pes. 

The  importance  of  the  assessments  of  the  1990  pes  and  other  coverage 
evaluation  efforts  is  underscored  by  the  Department’s  commitment,  as 
stated  in  the  guidelines,  to  complete  the  assessments  even  if  they  are  not 
finished  in  time  for  the  ac^justment  decision.  We  agree  that  the  Bureau 
should  complete  its  coverage  evaluation  activities  even  if  they  will  not 


^Federal  Formula  Programs:  Outdated  Population  Data  Used  to  Allocate  Most  Funds  (GAO/ 
GOD-90- 145,  Sept.  1^). 
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be  done  by  July  15, 1991.  The  results  could  be  used  for  future  census 
planning  by  identifying  improvements  needed  in  census-taking  proce¬ 
dures  and  future  acyustment  activities.  The  completed  assessments,  by 
identifying  the  estimated  error  in  the  census,  also  would  be  helpful  to 
preparing  estimates  made  between  censuses.  Such  estimates  are  used  to 
distribute  billions  of  dollars  to  state  and  local  governments. 
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The  PES  methodology  planned  for  the  1990  census  consists  of  two  parts: 
the  survey  and  the  resulting  mathematical  procedures.  The  survey 
includes  taking  an  independent  sample  of  the  population  and  comparing 
(matching)  the  information  to  census  records.  In  this  process,  the 
Bureau  does  field  canvassing  and  computerized  and  clerical  matching  of 
the  records.  Using  the  results  of  this  matching,  the  Bureau  will,  based  on 
statistical  principles,  employ  formulas  to  develop  coverage  error  esti¬ 
mates  for  various  population  groups  and  then  project  the  results 
nationally. 


PES  Design  and 
Operations 


For  the  1990  PES,  the  Bureau  took  a  stratified  random  sample  of  blocks.' 
The  sample  was  designed  to  be  representative  with  respect  to  type  of 
place  (large  metropolitan  areas,  suburban  areas,  rural  areas);  racial  and 
ethnic  composition;  tenure  (whether  residents  predominantly  own  or 
rent  their  housing  unit);  ana  other  variables.  The  pes  sample  for  1990 
consisted  of  about  170,000  housing  units  of  the  approximately  100  mil¬ 
lion  units  nationwide  and  5,000  block  clusters  of  the  approximately  6 
million  census  blocks  nationwide.^ 


The  Bureau  designed  its  pes  sample  with  the  intention  of  providing  suf¬ 
ficient  precision  for  estimates  of  the  “true  population”  for  various  sub¬ 
groupings  of  the  population  referred  to  as  post-strata.  Thus,  by 
comparing  the  estimated  true  population  to  the  census  count  for  each 
post-strata,  the  Bureau  would  be  able  to  estimate  the  over-  or 
undercounts.  For  1990,  the  Bureau  plans  to  develop  estimates  for  about 
1 ,400  post-strata.  The  variables  used  to  define  the  post-strata  include 
the  geographic  location;  place  and  size;  race/origin;  age  and  sex;  and,  in 
some  cases,  tenure  (household  owner  or  renter).  For  example,  one  post¬ 
strata  consists  of  black  males,  aged  0-9,  in  the  New  York  City  area. 


Field  Interviewing  February  and  March  1990,  experienced  Bureau  staff  members  visited 

each  sampled  block  and  were  instructed  to  record  the  addresses  of  all 
housing  units.  Starting  in  June  1990,  pes  interviewers  were  instructed  to 
visit  each  sampled  housing  unit  to  do  a  personal  interview  with  a  house¬ 
hold  member  and  to  complete  a  pes  questionnaire.  According  to  the 
Bureau’s  procedures,  the  pes  interviewers  attempted  to  complete  all 


'  In  urban  areas,  a  block  is  usually  equal  to  one  city  block.  In  suburban  and  rural  areas,  a  block 
usually  begins  and  ends  at  some  physical  or  geographic  feature,  like  a  road,  river,  or  county  boundary 
line.  The  Bureau  has  dividini  the  country  into  about  6  million  cen-sus  blocks. 

block  du-ster  is  either  <me  block  or  a  cluster  of  .several  small  blocks. 
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interviews  with  a  household  member.  However,  Bureau  instructions  said 
that  if  this  could  not  be  accomplished  after  repeated  visits  on  different 
days  and  at  different  times,  the  interviewers  were  permitted  to  inter¬ 
view  persons  who  were  not  household  members,  such  as  neighbors  and 
apartment  managers. 

During  the  interview,  the  interviewers  obtained  information  similar  to 
the  basic  data  obtained  during  the  census.  For  example,  they  asked  the 
name  and  characteristics  (sex,  age,  and  race)  of  each  person  living  at  the 
sampled  address  and  whether  the  occupants  own  or  rent  the  housing 
unit.  Additionally,  special  questions  were  asked  to  determine  where 
each  person  in  the  household  was  on  or  about  Census  Day.  Information 
on  others  who  might  have  lived  at  the  address  on  Census  Day  but  no 
longer  resided  at  the  sampled  address  at  the  time  of  the  interview  was 
also  obtained. 


Computer  and  Manual 
Matching 


As  they  were  completed,  pes  interview  questionnaires  were  sent  from 
field  offices  to  processing  offices  for  keying  the  pes  information  into 
computer  files.  The  address  keyed  for  each  person  was  the  address 
where  the  person  reportedly  lived  on  Census  Day. 

PES  and  census  computer  files  were  then  generated  for  the  computer 
matching  operation.®  Using  the  information  in  the  pes  and  census  files, 
the  computer  assigned  one  of  various  codes  to  each  person  on  each  pes 
block  in  the  file.  For  example,  codes  might  be  “M”  (match  to  both  files), 
“P”  (possible  match  to  both  files),  “N”  (pes  nonmatch  with  the  census 
file),  “E”  (census  nonmatch  with  the  pes  file),  “J”  (incomplete  pes  infor¬ 
mation  to  attempt  a  match),  or  “K”  (incomplete  census  information  to 
attempt  a  match). 

Clerks  in  the  processing  offices  reviewed  the  nonmatch  persons  to  deter¬ 
mine  if  any  could,  in  fact,  be  matched.  They  reviewed  the  actual  census 
and  pes  questionnaires,  as  well  as  other  data  to  help  in  this  determina¬ 
tion.  The  questionnaires  or  other  sources,  such  as  census  maps,  might 
have  provided  additional  information  that  showed  that  persons  did,  in 
fact,  match.  Multiple  clerical  groups  separately  reviewed  each  nonmatch 
situation  and  made  match  determinations,  pes  technicians,  who  are  more 


'The  PES  file  is  the  data  keyed  from  the  PES  questionnaires.  The  census  file  contains  information 
obtained  during  the  census  for  hou.sing  units  recorded  as  being  on  the  PES  sampled  blocks  or  any 
surrounding  blocks. 
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highly  trained,  reviewed  all  cases  to  ai^udicate  differences  in  the  cler¬ 
ical  determinations. 


Field  Follow-Up 
Operations 


To  further  resolve  the  nonmatches,  the  Bureau  sent  interviewers  to 
revisit  housing  units  to  follow  up  on  conflicting,  insufficient,  or  missing 
data.  The  follow-up  information,  however,  does  not  always  resolve  the 
nonmatch  situations.  The  follow-up  interviewer  may  not  have  been  able 
to  interview  the  nonmatch  person  or  another  reliable  person.  The  person 
interviewed  may  be  the  nonmatch  person  or  a  reliable  respondent,  but 
the  person  may  not  remember  the  Census  Day  address  or  provide  any 
better  information  than  the  initial  interview.  These  cases  may  be  classi¬ 
fied  as  unresolved  and  are  taken  into  consideration  in  the  undercount 
estimation  formula. 


Mathematical 

Procedures 


Before  any  calculations  can  be  made  to  determine  the  estimated  true 
population  for  the  various  groupings,  missing  person  characteristics 
must  be  statistically  imputed  and  unresolved  cases  statistically  resolved. 
Missing  person  characteristics  such  as  tenure  (owner  or  renter),  race, 
sex,  and  age  are  statistically  imputed  from  data  reported  for  other 
household  members  or  reported  on  complete  questionnaires  from  similar 
households  in  the  geographic  area.  For  unresolved  cases,  the  Bureau 
uses  statistical  imputation  methodology,  which  incorporates  cases  into 
the  formula  as  matched,  correctly  counted,  or  erroneously  counted. 

The  Bureau  estimates  the  true  population  for  each  grouping  by  using  the 
matching  codes.  To  do  this,  the  Bureau  uses  a  methodology  called  dual 
system  estimation  (dse).  Using  this  methodology,  the  true  population 
can  be  estimated  by  comparing  the  number  of  people  counted  in  both  the 
PES  and  the  census  and  the  people  counted  in  only  the  pes  or  the  census. 

We  provide  an  illustration  of  dse  in  a  forthcoming  report  on  the  Depart¬ 
ment  of  Justice:^  Suppose  a  game  warden  is  interested  in  knowing  the 
total  number  of  bears  in  a  preserve.  On  the  first  day,  she  captures  100 
bears,  ties  a  ribbon  around  the  neck  of  each  bear,  and  releases  them.  On 
the  second  day,  she  again  captures  100  bears  and  observes  how  many  of 
them  have  ribbons  around  their  necks.  The  most  extreme  outcomes  are 
instructive:  If  all  of  the  bears  captured  on  the  second  day  have  ribbons, 
the  total  population  is  probably  small  (possibly  no  more  than  100).  If 


^U.S.  Attorneys:  Better  Models  Can  Reduce  Resource  Disparities  Among  Offices,  (forthcoming). 
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none  have  ribbons,  the  total  population  is  probably  large  (at  a  minimum 
no  less  than  200). 

Generally,  the  larger  the  number  of  ribboned  bears  that  are  recaptured 
(i.e.,  the  larger  the  number  of  “matches”)  relative  to  the  total  number 
captured  on  the  second  day,  the  smaller  is  the  inferred  number  of  bears 
in  the  preserve.  To  be  sure,  assumptions  are  required  for  the  validity  of 
any  such  inference.  If  the  ribbons  did  not  allow  all  of  the  recaptured 
bears  to  be  identified  (say,  because  some  bears  ate  their  ribbons),  the 
inference  might  be  faulty.  Alternatively,  if  ribboned  bears  were  more 
likely  to  be  captured  on  the  second  day  than  unribboned  bears  (say, 
because  the  ribbons  impeded  escape),  the  inference  might  be  faulty. 

The  example  illustrates  a  principal  assumption  of  dse:  whether  an  indi¬ 
vidual  is  counted  in  the  pes  is  not  conditioned  by  whether  the  individual 
was  counted  in  the  census.  However,  violations  of  this  assumption, 
called  “correlation  bias,”  are  plausible.  If  certain  people — such  as  those 
who  wish  to  avoid  being  counted — are  likely  to  be  omitted  from  both 
the  PES  and  the  census,  then  dses  of  total  population  size  are  too  low 
(“biased  downward”)  and  estimated  coverage  probabilities  are  too  high 
(“biased  upward”). 

In  other  words,  the  capture/recapture  methodology  assumes  that 
chances  of  being  found  in  the  pes  are  not  affected  by  the  person’s  being 
counted  in  the  census.  However,  this  is  not  the  case.  The  Bureau’s  demo¬ 
graphic  analyses  have  demonstrated  the  existence  of  correlation  bias, 
that  is,  persons  missed  in  the  census  are  more  likely  to  be  missed  in  the 
PES  than  those  counted  in  the  census.  As  a  result,  the  undercount  esti¬ 
mates  from  the  pes,  while  important,  could  understate  the  true  census 
undercount. 

Using  the  estimated  true  population  for  each  grouping  derived  from  the 
DSE,  the  Bureau  calculates  the  estimated  census  error  for  that  grouping, 
or  the  percent  the  grouping  was  over-  or  undercounted.  The  over-  and 
underestimates  can  then  be  applied  to  each  block  in  the  nation.  For 
example,  if  the  adjustment  factor  for  black  males  aged  0-9  living  in  cen¬ 
tral  cities  in  the  Mid- Atlantic  area  is  1.02,  then  for  every  100  such 
people  counted  in  the  census  in  those  areas  two  persons  will  be  added.  If 
the  block  has  only  25  such  persons,  a  half  person  would  theoretically  be 
added.  Since  a  portion  of  a  person  is  unacceptable,  one  person  will  be 
added  using  a  statistical  rounding  procedure.  If  there  are  no  people  with 
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those  characteristics  enumerated  in  the  block  in  the  census,  none  will  be 
added.^ 


®This  illustration  is  from  The  1990  Post  Enumeration  Survey:  An  Overview,  by  Howard  Hogan, 
Bureau  of  the  Census,  presented  to  the  Population  Association  of  America  Annual  Meeting,  Toronto, 
Ontario,  May  3-6, 1090. 
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In  our  review  of  375  households  in  the  dress  rehearsal,  selected  by  ran¬ 
domly  sampling  10  percent  of  all  households  in  which  one  or  more  per¬ 
sons  were  not  matched  in  the  comparison  of  pes  and  census-counted 
persons  in  the  three  dress  rehearsal  sites,  we  found  examples  that  illus¬ 
trate  respondent  hostility  and  the  receipt  of  conflicting  information  that 
the  Bureau  encounters  in  doing  a  pes. 


Respondent  Hostility 


Example:  The  enumerator  reported  at  the  time  of  the  pes  that  the 
respondent,  a  grandmother,  was  hostile,  suspicious,  and  nonresponsive 
in  the  interview.  However,  the  enumerator  obtained  some  information 
about  the  household  from  the  woman’s  5-year-old  grandchild.  The 
grandmother  had  also  refused  to  provide  complete  information  for  the 
census  questionnaire  when  contacted  by  an  enumerator  during  the 
census  nonresponse  follow-up  operation.  The  census  enumerated  a  hus¬ 
band  and  wife.  The  pes  captured  a  woman  and  grandchild.  The  pes 
follow-up  confirmed  the  husband  was  correctly  enumerated.  For  this 
household,  the  pes  considered  two  persons  (husband  and  wife)  to  be  cor¬ 
rectly  enumerated  in  the  census  and  one  person  (child)  missed  in  the 
census.  If  the  grandchild  was  not  a  resident  of  the  household,  as  the 
census  classified  the  child,  the  true  population  was  overstated  and  the 
undercount  was  accordingly  overstated.  Because  of  the  hostility  of  the 
grandmother,  we  believe  the  number  of  persons  actually  residing  in  the 
housing  unit  on  Census  Day  is  uncertain. 


Inconsistent  and 

Conflicting 

Information 


Example  1:  The  same  respondent  for  the  census,  the  pes,  and  the  follow¬ 
up  provided  different  information  each  time  about  a  member  of  the 
household.  On  an  enumerator-completed  census  questionnaire,  the 
respondent  did  not  list  the  person;  in  the  pes,  the  respondent  reported 
the  person  had  been  at  the  housing  unit  for  5  years;  and  in  the  evalua¬ 
tion  follow-up,  the  respondent  did  not  know  where  the  person  lived  on 
dress  rehearsal  Census  Day.  The  person  was  classified  as  not  counted  in 
the  census.  That  classification  contributed  directly  to  a  higher  estimated 
net  undercount.  However,  if  as  a  result  of  the  pes  the  Bureau  could  not 
definitively  determine  where  the  person  should  have  been  counted  on 
Census  Day,  the  person  should  be  classified  as  “unresolved.”  That  clas¬ 
sification  could  result  in  a  lower  estimate  of  undercount  than  the  classi¬ 
fication  of  not  counted. 

Example  2:  Two  knowledgeable  respondents  provided  different  informa¬ 
tion  about  the  same  household  member.  In  the  pes,  the  mother  said  she 
did  not  remember  if  her  daughter  resided  at  the  housing  unit  or  her  own 
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apartment  on  Census  Day.  She  thought  she  listed  her  daughter  on  the 
census  questionnaire  because  she  believed  her  daughter  would  probably 
not  complete  the  questionnaire.  However,  the  daughter  was  not  listed  on 
the  census  questionnaire  completed  by  the  mother.  In  the  pes  follow-up, 
the  father  said  his  daughter  was  at  the  household  for  several  years, 
including  dress  rehearsal  Census  Day.  The  daughter  was  classified  as 
not  counted  in  the  census.  That  classification  contributed  directly  to  a 
higher  estimated  net  undercount.  If  the  daughter  should  have  been 
counted  by  the  pes  at  her  own  apartment  on  Census  Day,  and  was 
counted  there,  then  the  daughter  would  have  been  classified  as  correctly 
enumerated  and  would  have  supported  the  census  count  as  the  true 
population. 

Example  3:  Three  persons  were  classified  as  erroneously  enumerated  in 
the  census.  We  believe  the  circumstances  surrounding  this  case  are  con¬ 
fusing  and  could  support  different  determinations.  Both  the  census  and 
PES  information  were  obtained  in  July  1988.  The  census  respondent,  a 
household  member,  reported  the  three  persons  on  an  enumerator-com¬ 
pleted  census  questionnaire,  but  the  pes  respondent,  another  household 
member,  did  not.  The  pes  follow-up  interview  in  January  1989  reported 
the  three  persons  had  moved  in  February  1988,  about  1  month  before 
the  dress  rehearsal  Census  Day.  The  pes  follow-up  respondent  was  not 
identified.  The  evidence  seems  clear  that  the  three  persons  were  not  at 
the  household  at  the  time  of  the  pes  interview.  However,  whether  or  not 
they  were  at  the  household  on  Census  Day  is  debatable  on  the  basis  of 
the  available  information.  The  length  of  time  after  Census  Day  that  the 
information  was  obtained  causes  us  to  wonder  if  the  respondents 
remembered  exactly  when  the  three  persons  moved. 

This  “erroneous  enumeration”  classification  reduced  the  net  undercount 
rate.  If  these  three  persons  had  been  at  the  household  on  Census  Day 
and  had  been  classified  as  “correctly  enumerated”  in  the  pes,  the  census 
count  would  have  been  sustained  by  the  pes. 
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